High definition (HD) video has become widely available at both consumer and enterprise levels. In the home entertainment industry, for instance, there is a strong consumer interest in the new Ultra HD/4K video technology. However, higher resolution projectors are costly. Alternatively, super-resolution is an image processing research domain, which seeks to produce higher resolution imagery in low resolution contexts.
For example, image and video content can be acquired using cameras at very high resolutions while modern display projectors are very limited in display resolution by comparison. As a result, there is a need to increase the apparent display resolution of projectors. A typical model to enhance the resolution of a projector consists of decomposing the high-resolution signal to multiple lower resolution signals and displaying them with different offsets and, in the case of video, at a higher frame rate.
The super-resolution (SR) problem of producing higher resolution imagery in low resolution contexts is well known in the literature, and different methodologies have been proposed to address this problem.
One class of SR methods uses Fourier Transforms to solve the SR problem. Frequency based SR methods rely on three fundamental principles, 1) the input image is band limited, 2) there is an aliasing relation between continuous Fourier transform (CFT) and discrete Fourier transform (DFT), 3) there exists a shifting property of the Fourier transform. The aliasing relation between CFT and DFT is solved using the least square approach. Frequency-based SR models tend to be sensitive to model errors. Additionally, such models handle translational motions, but non-translation motion significantly degrades the model performance. Moreover, frequency based SR methods are limited to space invariant degradation models, which limits their performance in many real world scenarios where the degradation model varies spatially (e.g. spatially variant point spread function (PSF)). Generally, frequency based SR methods suffer from the limited ability to include spatial domain a-priori knowledge for problem regularization.
Another class of super-resolution methods solves the problem in the spatial domain. Spatial domain SR approaches can accommodate more complex SR-related issues such as global and local motion, spatially varying PSF, motion blur, compression artifacts and more. Examples of spatial domain SR methods include iterative back-projection, non-iterative spatial domain and hybrid method. Stochastic methods, especially Bayesian-based methods, where the SR problem is looked at as a statistical estimation problem, have rapidly gained attention of researchers in the SR field due to their ability to include a-priori constraints (e.g. edge-preserving image prior) to achieve satisfactory solutions of the SR problem. On the other hand, spatial domain based methods are computationally expensive and hence are not suitable for real time applications. Specifically, the inclusion of a-priori constraints is not easily achieved in iterative back-projection related methods. Even stochastic methods, which are known to be a flexible and convenient way to model a priori SR constraints, have the limitation of being unable to reconstruct the high frequency components of images very well.
Additionally, most SR methods (both frequency and spatial based) are time consuming, and hence cannot meet the real-time constraints of SR based applications.
Different image restoration techniques have been developed to correct for optical aberration and recover an approximation of the original image. Conventional methods mainly involve a transformation of the source image by a filter prior to display. However, in most existing approaches, the filtering operation is carried out in the frequency domain which requires complex hardware. Due to the limited processing power in commercial projectors, it is not feasible to achieve frequency-domain optical aberration correction in a real-time implementation.